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ABSTRACT 

In this paper, we study the performance of IPOP- s *aACM- 
ES, recently proposed self-adaptive surrogate-assisted Co- 
variance Matrix Adaptation Evolution Strategy. The algo- 
rithm was tested using restarts till a total number of function 
evaluations of 10 6 D was reached, where D is the dimension 
of the function search space. The experiments show that the 
surrogate model control allows IPOP- s *aACM-ES to be as 
robust as the original IPOP-aCMA-ES and outperforms the 
latter by a factor from 2 to 3 on 6 benchmark problems with 
moderate noise. On 15 out of 30 benchmark problems in di- 
mension 20, IPOP- s *aACM-ES exceeds the records observed 
during BBOB-2009 and BBOB-2010. 

Categories and Subject Descriptors 

G.1.6 [Numerical Analysis]: Optimization — global opti- 
mization, unconstrained optimization; F.2.1 [Analysis of 
Algorithms and Problem Complexity]: Numerical Al- 
gorithms and Problems 

General Terms 

Algorithms 

Keywords 

Benchmarking, black-box optimization, evolution strategy, 
CMA-ES, self-adaptation, surrogate models, ranking sup- 
port vector machine, surrogate-assisted optimization 

1. INTRODUCTION 

When dealing with expensive optimization objectives, the 
surrogate-assisted approaches proceed by learning a surro- 
gate model of the objective, and using this surrogate to re- 
duce the number of computations of the objective function 
in various ways. 

Many surrogate modelling approaches have been used within 
Evolution Strategies (ESs) and Covariance Matrix Adapta- 
tion Evolution Strategy (CMA-ES): Radial Basis Functions 



network [8], Gaussian Processes |17| . Artificial Neural Net- 
work [3], Support Vector Regression [12] , Local- Weighted 
Regression [111 IT], Ranking Support Vector Machine (Rank- 
ing SVM) [16] [13] [9]. In most cases, the surrogate model 
is used as a filter (to select Xp re promising pre-children) 
and / or to estimate the fitness of some individuals in the cur- 
rent population. An example of surrogate-assisted CMA-ES 
with filtering strategy can be found in [13| . 

A well-known drawback of surrogate-assisted optimization 
is a strong dependence of the results on hyper-parameters 
used to build the surrogate model. Some optimal settings 
of hyper-parameters for a specific set of problems can be 
found by offline tuning, however for a new problem they are 
unknown in the black-box scenario. Moreover, the optimal 
hyper-parameters may dynamically change during the opti- 
mization of the function. 

Motivated by this open issues, new self-adapted surrogate- 
assisted s *aACM-ES algorithm have been proposed combin- 
ing surrogate-assisted optimization of the expensive func- 
tion and online optimization of the surrogate model hyper- 
parameters [14] , 

2. THE ALGORITHMS 

2.1 The (n/n w , A)-CMA-ES 

In each iteration t, (fi/fi w , A)-CMA-ES 6 samples A new 
solutions Xi G R 13 , where i = 1, . . . , A, and selects the best 
\i among them. These \i points update the distribution of 
parameters of the algorithm to increase the probability of 
successful steps in iteration t + 1. The sampling is defined 
by a multi-variate normal distribution, A/"(m*, <r' 2 C'), with 
current mean of distribution m', D x D covariance matrix 
C' and step-size <x* . 

The active version of the CMA-ES proposed in [7j [10] in- 
troduces a weighted negative update of the covariance ma- 
trix taking into account the information about A — fi worst 
points as well as about /j, best ones. The new version im- 
proves CMA-ES on 9 out of 12 tested unimodal functions by 
a factor up to 2, and the advantages are more pronounced 
in larger dimension. While the new update scheme does not 
guarantee the positive-definiteness of the covariance matrix, 
it can be numerically controlled [TJ. Since in our study we 
do not observe any negative effects of this issue, we will use 
aCMA-ES, the active version of the CMA-ES, for compari- 
son with the surrogate-assisted algorithms. 
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2.2 The ACM-ES 

The "ACM-ES [14] is the surrogate-assisted version of 



the (fi/fMp, A)-CMA-ES, where the surrogate model is used 
periodically instead of the expensive function for direct op- 
timization. The use of Ranking SVM allows to preserve 
the property of CMA-ES of invariance with respect to rank- 
preserving transformation of the fitness function. The prop- 
erty of invariance with respect to the orthogonal transforma- 
tion of the search space is preserved thanks to the definition 
of the kernel function by the covariance matrix, adapted 
during the search. 

In S *ACM-ES we perform the following surrogate-assisted 
optimization loop: we optimize the surrogate model / for h 
generations by the CMA-ES, then we continue and optimize 
the expensive function f(x) for one generation. To adjust 
the number of generations h for the next time, the model 
error can be computed as a fraction of incorrectly predicted 
comparison relations that we observe, when we compare the 
ranking of the last A evaluated points according to f(x) and 
/. The S *ACM-ES uses the generation of the CMA-ES as 
a black-box procedure, and it has been shown in [14], that 
the improvement of the CMA-ES from passive to active ver- 
sion (aCMA-ES) leads to a comparable improvement of its 
surrogate-assisted versions ("ACM-ES and s *aACM-ES). 

The main novelty of the S *ACM-ES is the online optimiza- 
tion of the surrogate model hyper-parameters during the op- 
timization of the fitness function. The algorithm performs 
the search in the space of model hyper-parameters, generat- 
ing Xhyp surrogate models in each iteration. The fitness of 
the model can be measured as a prediction error of the rank- 
ing on the last A evaluated points. This allows the user to 
define only the range of hyper-parameters and let algorithm 
to find the most suitable values for the current iteration t. 

The detailed description of S *ACM-ES is given in [14]. 

2.3 The Benchmarked Algorithms 

For benchmarking we consider S *ACM-ES in IPOP restart 
scenario (IPOP- s *aACM-ES) using default parameters and 
termination criteria as given in [7] and [14]. The only one 
parameter of the surrogate part of IPOP- s *aACM-ES differ- 
ent from the default one is the index of generation g s tart 
when we start to use the surrogate model. For noisy case 
we set g s tart = 5(i re start + 1) instead of default g ata rt = 10. 
We found that sometimes it makes sense to postpone the 
surrogate-assisted search if several restarts (irestart) were 
performed. 

3. RESULTS 

Results from experiments according to 4j on the bench- 
mark functions given in [2] [5] are presented in Figures [3] and 
[4] and Figure [2] The expected running time (ERT), 
used in the figures and table, depends on a given target 
function value, ft = / op t + A/, and is computed over all 
relevant trials (on the first 15 instances) as the number of 
function evaluations executed during each trial while the 
best function value did not reach ft, summed over all trials 
and divided by the number of trials that actually reached ft 
[4][15]. Statistical significance is tested with the rank- 
sum test for a given target A/t using, for each trial, either 
the number of needed function evaluations to reach A/t 
(inverted and multiplied by —1), or, if the target was not 
reached, the best A/-value achieved, measured only up to 
the smallest number of overall function evaluations for any 
unsuccessful trial under consideration if available. Tables [1] 
and [2] give the Expected Running Time (ERT) for targets 



10 1 ' - 1 .- 3 . - 5 . - 7 divided by the best ERT obtained during 
BBOB-2009 (given in the ERT bcst row), respectively in 5-D 
and 20-D. Bold entries correspond to the best (or 3-best if 
there are more than 3 algorithms) values. The median num- 
ber of conducted function evaluations is additionally given 
in italics, if ERT(10~ 7 ) = oo. #succ is the number of trials 
that reached the final target / opt + 10 -8 . Entries with the J, 
symbol are statistically significantly better (according to the 
rank-sum test) compared to the best algorithm in BBOB- 
2009, with p — 0.05 orp= 10~ fe where k > 1 is the number 
following the 4- symbol, with Bonferroni correction of 30. 

The IPOP- s *aACM-ES outperforms IPOP- s *aACM-ES usu- 
ally by a factor from 2 to 3 on functions with moderate 
noisy. This is the case for Sphere (/ioi,/i02,/io3) and Rosen- 
brock (/io4,/i05,/i06) functions with Gaussian, Uniform and 
Cauchy noise models. It seems that the moderate noise only 
slightly affects the quality of the surrogate model and allows 
to have a speedup comparable to one of the noise-less case. 

On most functions with severe noise the surrogate model 
usually is not used (n oscillates around zero), because it 
gives a very imprecise prediction of the expensive function. 
On 20-dimensional /124 Schaffer function with Cauchy noise 
IPOP- s *aACM-ES requires about 7 times more functions 
evaluations to reach the optimum with Af opt = 10~ 7 than 
IPOP-aCMA-ES (see Fig. [2j. However, according to Ta- 
ble [2] the performance for Af op t = 10 -5 is exactly the 
same for both algorithms, therefore, we suppose that the 
loss of performance for Af opt = 10~ 7 can be explained by 
some influence of surrogate-assisted search on restart condi- 
tions. We also found that if use default coefficient ci and c M 
for the covariance matrix update instead of noisy settings 
of ci/5 and c M /5, then IPOP- s *aACM-ES performs as well 
as IPOP-aCMA-ES. However, the use of default coefficients 
worsen the results of IPOP-aCMA-ES and IPOP-"*aACM- 
ES on other problems. 

We also observe some loss in performance of /125 Griewank- 
Rosenbrock and /128 Gallagher functions for d < 5. For /115 
we observe the speedup for d — 5 and loss for d = 20, prob- 
ably because of the same reasons as for /124. 

The IPOP- s *aACM-ES improves (sometimes insignificantly) 
the records in dimension 20 on f 10 i, /102, /103, /104, /105, 

/l06, /l07, /l09, ,fll2, /ll4, /ll7, /ll8, /l20, /l22, /l23, /l27, 
/l29- 

4. CPU TIMING EXPERIMENT 

For the timing experiment the IPOP- s *aACM-ES was run 
on noiseless /1, fg, fio and /15 functions without self-adaptation 
of surrogate model hyper-parameters. The crucial hyper- 
parameter for CPU time measurements, the number of train- 
ing points was set Ntrammg = [40 + 4D 1,7 J as a function of 
dimension D [14] , 

These experiments have been conducted on a single core 
with 2.4 GHz under Windows XP using Matlab R2006a. 

On uni-modal functions the time complexity of surrogate 
model learning increases cubically in the search space di- 
mension (see Fig. [1]) and quadratically in the number of 
training points. For small dimensions (D < 10) the overall 
time complexity increases super-linearly in the dimension. 
The time complexity per function evaluation depends on 
the population size, because one model is used to estimate 
the ranking of all points of the population. This leads to a 
smaller computational complexity on multi-modal functions, 
e.g. /15 Rastrigin function, where the population becomes 
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Figure 1: CPU cost per function evaluation of 
IPOP-aACM-ES with fixed hyper-parameters. 

much larger after several restarts. The time complexity on 
noisy functions is more similar to the one of Rastrigin func- 
tion, because in both cases the large populations are used. 

The results presented here does not take into account the 
model hyper-parameters optimization, where \h yP surrogate 
models should be build at each iteration, which leads to an 
increase of CPU time per function evaluation by a factor of 
\ hyp . For IPOP- s *aACM-ES \ hyp was set to 20. 

5. CONCLUSION 

In this paper, we have compared the recently proposed 
self-adaptive surrogate-assisted IPOP-"*aACM-ES with the 
IPOP-aCMA-ES on noisy benchmark problems. The surrogate- 
assisted IPOP- s *aACM-ES algorithm outperforms the orig- 
inal IPOP-aCMA-ES by a factor from 2 to 3 on the func- 
tions with moderate noise, and usually performs not worse 
on other functions. The IPOP- s *aACM-ES algorithm im- 
proves the records on 15 out of 30 functions in dimension 
20. 
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Figure 2: Expected running time (ERT) divided by dimension for target function value 10~ 8 as log 10 values 
versus dimension. Different symbols correspond to different algorithms given in legend of /101 and /130. 
Light symbols give the maximum number of function evaluations from all trials divided by the dimension. 
Horizontal lines give linear scaling, the slanted dotted lines give quadratic scaling. Legend: o: IPOP-aCMA, 
V: IPOP-saACM 



A/ pt 


lei IcO le-1 lc-3 lc-5 lc-7 


#succ 


flOl 

IPOP-aC 
IPOP-sa 


11 37 44 62 69 75 
3.0(2) 3.0(1) 4.2(1) 5.8(2) 7.6(2) 9.3(2) 
1.9(1) 2.1(0.5)* 2.2(0.5)* 3 2.2(0-5)* 4 2.5(0.3)* 4 2.8(0.3)* 4 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 


1102 

IPOP-aC 
IPOP-sa 
A /opt 


11 35 50 72 86 99 
3.5(3) 3.4(1) 3.8(0.9) 5.0(0.6) 6.1(0.8) 7.0(0.7) 
2.6(2) 2.1(0.7) 2.0(0.5)* 4 1.9(0.4)* 4 2.1(0.3)* 4 2.2(0.3)* 4 
lcl leO lc-1 le-3 le-5 le-7 


15/15 
15/15 
15/15 


fl03 

IPOP-aC 
IPOP-sa 

A /opt 


11 28 30 31 35 115 
2.9(1) 4.1(1) 6.3(1) 11(2) 16(2) 6.5(0.6) 
3.0(1) 2.8(0.7)* 3.4(0.9)* 3 4.6(1)* 4 7.1(3)* 4 3.2(1)* 4 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl04 

IPOP-aC 
IPOP-sa 

A /opt 


173 773 1287 1768 2040 2284 
1.5(0.5) 2.0(2) 17(1) 1.6(1.0) 1.5(0.9) 1.4(0.8) 
0.88(0.3)* 1.5(1) 1-1(1) 0.89(0.8) 0.80(0.7)* 0.73(0.7)* 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl05 

IPOP-aC 
IPOP-sa 

A /opt 


167 1436 5174 10388 10824 11202 
2.8(0.5) 3.0(2) 1.3(0.8) 0.70(0.4) 0.70(0.4) 0.70(0.4) 
1.4(0.5) 1.5(2) 0.63(0.4) 0. 33(0. 2)£^D. 32(0. 2) J^0. 32(0. 2)£, 

lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 

#succ 


flOB 

IPOP-aC 
IPOP-sa 

A /opt 


92 529 1050 2666 2887 3087 
2.3(0.7) 1.9(0.6) 1.6(0.3) 0.87(0.1) 0.90(0.1) 0.92(0.1) 
1.7(0.9) 0.89(0. 2)*%. 60(0. 2)*%. 30(0.1) Jjo. 32(0.1) J^O. 36(0.1) J; 

lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 

#succ 


fl07 

IPOP-aC 
IPOP-sa 

A /opt 


40 228 453 940 1376 1850 
1.9(2) 2.8(4) 1.9(2) 1.4(1.0) 1.3(0.7) 1.4(0.7) 
1.4(2) 1.1(0.7) 1-4(1) 1.3(0.8) 1.4(1.0) 1-9(1) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fios 

IPOP-aC 
IPOP-sa 

A /opt 


87 5144 14469 30935 58628 80667 
11(17) 0.68(0.5) 0.64(0.2) 0.64(0.4) 0.66(0.3) 0.94(0.5) 
7.8(9) 1 1(1) 0.60(0.4) 0.80(0.5) 0.74(0.4) 1.0(0.6) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl09 

IPOP-aC 
IPOP-sa 

A /opt 


11 57 216 572 873 946 
3.1(2) 1.9(0.6) 1.1(0.3) 0.92(0.2) 1.2(0.2) 1.6(0.3) 
2.9(2) 1.6(0.6) 0.83(0.3) 1.2(0.3) 1.2(0.3) 1.6(0.4) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fllO 

IPOP-aC 
IPOP-sa 

A /opt 


949 33625 1.2c5 5.9c5 6.0c5 6.1c5 
0.54(0.7) 4.3(5) 3.2(3) 0.79(0.8) 0.79(0.8) 0.79(0.8) 
0.71(0.3) 5.6(7) 3.5(4) 0.78(0.8) 0.84(0.8) 0.84(0.8) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 


fill 

IPOP-aC 
IPOP-sa 

A /opt 


6856 6.1c5 8.8c6 2.3c7 3.1c7 3.1c7 
0.54(0.4) 7.3(8) 7.9(9) oo oo coSeil 
0.50(0.3) 7.4(8) 1.8(2) 1.00(1) 0.74(0.7) 0.74(0.6) 
lcl leO le-1 lc-3 lc-5 le-7 


3/15 
0/15 
3/15 
#succ 


fll2 

IPOP-aC 
IPOP-sa 

A /opt 


107 1684 3421 4502 5132 5596 
2.2(0.6) 1.2(0.6) 1.0(0.5) 1.1(0.3) 1.1(0.3) 1.1(0.3) 
1.4(0.6)* 2 1.5(2) 1.2(0.9) 1.2(0.6) 1.2(0.6) 1.2(0.6) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 


fll3 

IPOP-aC 
IPOP-sa 

A /opt 


133 1883 8081 24128 24128 24402 
4.4(6) 0.78(1) 0.77(0.7) 0.32(0.2) 0.32(0.2) 0.32(0.3) 
3.0(2) 0.82(0.8) 0.47(0.6) 0.35(0.4) 0.35(0.4) 0.35(0.4) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fll4 

IPOP-aC 
IPOP-sa 

A /opt 


767 14720 56311 83272 83272 84949 
3.3(3) 0.61(0.5) 0.49(0.3) 0.60(0.2) 0.60(0.2) 0.61(0.2) 
2.6(3) 0.73(0.4) 0.54(0.4) 0.44(0.2) 0.44(0.2) 0.44(0.3) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fll5 

IPOP-aC 
IPOP-sa 


64 485 1829 2550 2550 2970 
1.8(1.0) 0.81(0.9) 1.1(1) 1-4(1) 1-4(1) 1-5(1) 
1.6(0.6) 0.98(1) 12(1) 1.2(0.8) 1.2(0.8) 1.1(0.7) 


15/15 
15/15 
15/15 



A/opt 


lcl IcO lc-1 lc-3 lc-5 lc-7 


#succ 


file 

IPOP-aC 
IPOP-sa 

A/opt 


5730 14472 22311 26868 30329 31661 
0.88(0.8) 0.62(0.6) 0.57(0.4) 0.73(0.5) 0.68(0.4) 0.68(0.4) 
1.0(1) 0.74(0.8) 0.56(0.5) 0.59(0.5) 0.56(0.4)^ 0.56(0.4)^ 

lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 

#succ 


fll7 

IPOP-aC 
IPOP-sa 

A/ pt 


26686 76052 l.lc5 1.4c5 1.7c5 1.9c5 
0.65(0.5) 0.48(0.4) 0.44(0.2)^ 2 O.47(0.2) ^^0.43(0.2) ^20.46(0.2)^< 
0.60(0.5) 0.43(0.3) 0.42(0.3)^ 2 0-54(0.3)^ 0.59(0.3) 0.60(0.2) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fllS 

IPOP-aC 
IPOP-sa 

A/ op t 


429 1217 1555 1998 2430 2913 
1.7(0.6) 0.88(0.3) 0.91(0.2) 0.91(0.1) 0.91(0.1) 0.90(0.1) 
0.49(0.1)J 4 0.25(0.1)J 4 0.33(0.1) J|0. 48(0.1) J^. 63(0.1) J|0. 74(0. 3) J. 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 


fll9 

IPOP-aC 
IPOP-sa 

A/ pt 


12 657 1136 10372 35296 49747 
1.5(1) 0.55(0.6) 0.58(0.5) 0.55(0.4) 0. 49(0. 2)^ 2 0-61 (0.4) 
4.9(10) 1-1(2) 0.99(0.9) 0.50(0.5) 0.47(0.2) ^ 2 0. 61 (0.5) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 


fl20 

IPOP-aC 
IPOP-sa 

A/ OB t 


16 2900 18698 72438 3.3c5 5.5c5 
18(39) 0.76(1) 0.65(0.5) 0.83(0.5) 0.43(0.3)^ 2 0.66(0.5) 
26(56) 1 1(1) 0.80(0.5) 0.52(0.3) 0.36(0.3) j^. 64(0. 5) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fl21 

IPOP-aC 
IPOP-sa 

A/ pt 


8.6 111 273 1583 3870 6195 
3.0(3) 1.1(0.4) 1.1(0.3) 0.78(0.3) 0.84(0.1) 0.82(0.1)^ 
3.3(4) 0.98(0.5) 0.97(0.5) 1.0(0.6) 0.95(0.4) 0.86(0.2) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 

#BUCC 


fl22 

IPOP-aC 
IPOP-sa 

A/ pt 


10 1727 9190 30087 53743 l.lc5 
2.9(3) 1-4(1) 0.71(0.6) 0.48(0.2)^20.66(0.3) 0.58(0.5)^ 
2.6(3) 1.2(0.8) 0.75(0.5) 0.58(0.2)^ 0.77(0.3) 0.66(0.3)^ 
lcl IcO lc-1 le-3 le-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl23 

IPOP-aC 
IPOP-sa 
A/opt 


11 16066 81505 3.4c5 6.7c5 2.2c6 
28(60) 0.94(0.7) 0.66(0.4) 0.67(0.4) 0.56(0.2)^29.60(0.3) 
23(52) 0.74(0.4) 0.70(0.5) 0.58(0.3) 0.80(0.7) 0.71(0.5) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fl24 

IPOP-aC 
IPOP-sa 

A/ op t 


10 202 1040 20478 45337 95200 
2.6(3) 1.2(0.8) 2.1(3) 0.93(0.6) 0.94(0.5) 0.59(0.3) 
2.6(2) 1.3(0.6) 3.1(4) 0.85(0.7) 0.97(0.8) 4.1(8) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl25 

IPOP-aC 
IPOP-sa 

A/opt 


11 1 2.4c5 2.4c5 2.5c5 
1(0) 26(22) 3142(3751) 0.53(0.4) 0.57(0.4) 0.57(0.4) 
1.5(0) 18(18) 4308(4084) 0.56(0.6) 0.79(0.9) 4.8(6) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl26 

IPOP-aC 
IPOP-sa 

A/ op t 


11 1 CO DO OO 

1(0) 62(98) 7882(5858) 

1(0) 165(444) 9361(8720) 1.3e7(lc7)1.3e7(2e7)L.3e7(le7 
lcl leO le-1 le-3 lc-5 le-7 




0/15 
5/15 
#succ 


fl27 

IPOP-aC 
IPOP-sa 

A/ op t 


11 1 3.4c5 3.9c5 4.0c5 
1.1(0) 30(24) 3085(2864) 0.50(0.5) 0.46(0.4) 0.46(0.4) 
1(0) 20(22) 2432(3044) 0.30(0.3) 0.32(0.3)^0.33(0.3)^ 

lcl IcO le-1 le-3 le-5 le-7 


15/15 
15/15 
15/15 


fl28 

IPOP-aC 
IPOP-sa 

A/ pt 


111 4248 7808 12447 17217 21162 
2.6(1) 1.8(2) 47(2) 30(3) 22(2) 18(2) 
1.3(1.0) 16(21) 265(331) 203(342) 147(152) 124(212) 
lcl leO le-1 lc-3 lc-5 lc-7 


15/15 
14/15 
11/15 
#succ 


fl29 

IPOP-aC 
IPOP-sa 

A/ op t 


64 10710 59443 2.8c5 5.1c5 5.8c5 
11(16) 10(5) 16(27) 7.0(8) 3.9(7) 3.5(6) 
9.1(12) 10(21) 19(42) 13(18) 7.1(10) 6.3(9) 
lcl leO le-1 le-3 le-5 lc-7 


15/15 
10/15 
9/15 


fl30 

IPOP-aC 
IPOP-sa 


55 812 3034 32823 33889 34528 
1.4(0.8) 143(288) 391(625) 36(46) 35(45) 35(44) 
1.4(0.9) 27(27) 126(131) 42(76) 41(74) 40(72) 


10/15 
11/15 
13/15 



Table 1: Expected running time (ERT in number of function evaluations) divided by the respective best ERT 
measured during BBOB-2009 (given in the respective first row) for different Af values in dimension 5. The 
central 80% range divided by two is given in braces. The median number of conducted function evaluations 
is additionally given in italics, if ERT(1CP 7 ) = oo. #succ is the number of trials that reached the final target 
/opt + 10~ 8 . Best results are printed in bold. 



A/opt 


lei leO le-1 le-3 le-5 le-7 


#succ A/ op t 


lcl IcO lc-1 lc-3 lc-5 lc-7 


#succ 


flOl 

IPOP-aC 
IPOP-aa 

A /opt 


59 425 571 700 739 783 
6.1(1.0) 1.5(0.1) 1.6(0.1) 2.1(0.2) 2.7(0.1) 3.3(0.2) 
4.7(1)* 0. 86(0. 2) t4 0. 73(0. l)||o. 78(0. 1) J^D. 88(0. 1)^0. 99(0.1)*' 
lei leO le-1 lc-3 le-5 le-7 


15/15 file 
15/15 IPOP-aC 
15/15 IPOP-aa 

#aucc A/ pt 


5.01:5 6.9c5 8.9c5 1.0c6 l.lc6 l.lc6 
0.36(0.1) 0.28(0. 1)_|_ 0.23(0.0)^.22(0.0)^0.23(0.0)^0.24(0.0)^ 
0.46(0.1) 0.35(0.1) 0.30(0.1) 0.28(0. 1)^ 4 0. 29(0. 1)^ 4 0. 30(0. 1) ± 4 
lcl IcO lc-1 lc-3 lc-5 lc-7 


1 5 / 1 5 
15/15 
15/15 
#succ 


fl02 

IPOP-aC 
IPOP-aa 

A /opt 


231 399 579 921 1157 1407 
1.5(0.4) 1.6(0.2) 1.6(0.2) 1.6(0.1) 1.7(0.1) 1.8(0.1) 
1.2(0.2) 0. 93(0. 2) * 4 0. 75(0. l)||o. 61(0. 1)| Jo. 60(0.0) J p. 60(0.0) *j 
lei leO le-1 lc-3 le-5 le-7 


15/15 fll7 
15/15 IPOP-aC 
15/15 IPOP-aa 

#aucc A/ op t 


1.8c6 2.5c6 2.6c6 2.9c6 3.2c6 3.6c6 
0. 37(0. 2)^ 2 0. 35(0. 2) ^ 4 0. 41(0. 3)^ 3 0. 45(0. 2) ^ 3 0. 47(0. 2)^ 3 0. 56(0. 3)^3 
0. 29(0. 1)^30. 26(0. 1)^ 4 0. 29(0. 1)^ 4 0.35(0. 2) ^ 4 0. 42(0. 2) ^30. 48(0.4)^ 2 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


IPOP-aC 
IPOP-aa 

A /opt 


65 417 629 1313 1893 2464 

5.4(0.9) 1.5(0.1) 1.5(0.1) 1.2(0.1) 1.2(0.1) 1.2(0.1) 

3. 9(0. 8)* 2 0. 83(0. 2) £^0. 66 (O.l)*^. 52 (0.1) |^0. 55(0.1) *^.58(0.1)*j 

lcl leO lc-1 lc-3 le-5 le-7 


14/15 
15/15 IPOP-aC 
15/15 IPOP-aa 
#succ A/ pt 


01)08 11786 17514 26342 30062 32650 
0.82(0.1) 0.78(0.2) 0.71(0.2) 0.63(0. 1) ^ 3 0. 64(0. 1)^ 4 0. 66(0. 1) ^3 
0. 42 (0. 2) *^. 49 (0.1)*^. 54(0.1) £30. 54(0. 2) ^ 4 0. 56(0. 1)^ 4 0. 58(0.1)*^ 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fl04 

IPOP-aC 
IPOP-sa 

A /opt 


23690 85656 1.7c5 1.8c5 1.9c5 2.0c5 
4.9(3) 1.6(1.0) 0.82(0.5) 0.80(0.5) 0.79(0.5) 0.77(0.4) 
3.5(2) 1.1(0.6) 0.55(0.3) 0.52(0.3) 0.50(0.3) 0.49(0.3) 
lei leO lc-1 lc-3 lc-5 le-7 


15/15 fll9 
15/15 IPOP-aC 
15/15 IPOP-aa 
#succ A/ pt 


2771 29365 35930 4.1c5 1.4c6 1.9c6 
1.6(1) 0.62(0.4) 0.83(0.4) 0. 42(0. 3)^ 2 0. 25(0. 1) ^ 4 0. 28(0. 1) ^ 
1.5(2) 0.74(0.3) 0.88(0.3) 0.33(0. 1) ^ 3 0. 25(0. l)j. 4 0. 37(0. 3) ^3 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
15/15 
#succ 


fl05 

IPOP-aC 
IPOP-sa 

A /opt 


1.9c5 6.1c5 6.3c5 6.5c5 6.6c5 6.7c5 
1.2(0.3) 0. 43(0. 1)^ 3 0. 43(0. 1)^ 3 0. 44(0. 1)^ 3 0. 44(0. 1)^ 3 0. 44(0. 1)^3 
1.0(0.4) 0.36(0.1)^30.35(0.1)^30.34(0.1)^30.34(0.1)^30.33(0.1)^; 
lei leO lc-1 lc-3 lc-5 le-7 


15/15 fl20 
15/15 IPO P-aC 
15/15 IPOP-aa 
#aucc Af 


36040 1.8c5 2.8c5 1.6c6 6.7c6 1.4c7 
0.62(0.3) 0.64(0.2) 0.79(0.4) 0.78(0.4) 0.44(0-2)^ 3 0.40(0.2) ^3 
0.65(0.5) 0.80(0.4) 1.2(0.6) 0.87(0.5) 0.42(0.1)^30.39(0.2)^; 
lcl leO le-1 lc-3 lc-5 le-7 


13/15 
15/15 
15/15 
#aucc 


flOB 

IPOP-aC 
IPOP-aa 

A /opt 


11480 21668 23746 25470 26492 27360 
0.77(0.2) 0. 76(0. 2)^ 3 0. 84(0. 2)^ 2 0. 89(0.1) 0.91(0.1) 0.91(0.1) 
0.28(0.1)^^0.37(0.3) 1^.39(0.2) 1^3.45(0.2) ||o. 46(0. 2) ||o.47(0.2)^ 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 fl2 i 
15/15 IPOP-aC 
15/15 IPOP-aa 
#aucc A/ op t 


249 769 1426 9304 34434 57404 
1.2(0.3) 1.0(0.2) 1.1(0.2) 0.77(0.1)^ 3 0. 56(0. 1)^ 4 0. 64(0. 1)^, 
1.2(0.4) 0.85(0.2) 0.92(0.3) 0.69(0. l)i 3 0. 58(0. l)i 4 0. 71 (0. 1) 13 
lcl 1 eO 1©- 1 le-3 le-o le- Y 


15/15 
15/15 
15/15 

#3UCC 


fl07 

IPOP-aC 
IPOP-sa 

A /opt 


8571 13582 16226 27357 52486 65052 
0.81(0.4) 0.97(0.4) 1.3(0.6) 1.5(0.9) 1.3(0.7) 1.1(0.5) 
0.83(0.5) 1.3(0.8) 1.4(0.7) 1.3(0.8) 1.0(0.5) 0.94(0.4) 
lcl leO lc-1 lc-3 lc-5 le-7 


15/15 fl22 
15/15 IPOP-aC 
15/15 IPOP-aa 

#aucc A/ p t 


692 52008 1.4c5 7.9c5 2.0c6 5.8c6 
2.3(3) 0.81(0.3) 0.97(0.6) 0.56(0.3) 0.66(0.1) 0.81(0.9) 
2.5(3) 0.94(0.6) 0.81(0.3) 0.53(0.1)^ 0.51(0.2)^ 0.57(0.3) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
15/15 
#succ 


fios 

IPOP-aC 
IPOP-sa 

^ J opt 


58063 97228 2.0c5 4.5c5 6.3c5 9.0c5 
0.74(0.3) 0.98(0.5) 1.0(0.6) 1.1(0.6) 1.5(0.4) 1.5(0.3) 
0.72(0.3) 1.2(0.6) 1.1(0.6) 1.2(0.7) 1.6(0.3) 1.4(0.3) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 fl23 
15/15 IPOP-aC 
15/15 IPOP-aa 
#aucc A/ pt 


1063 5.3c5 1.5c6 5.3c6 2.7c7 1.6c8 
6.4(4) 0.72(0.4) 0.88(0.8) 0.94(0.7) 0.50(0.1) 1.9(2) 
7.6(8) 0.77(0.5) 0.81(0.5) 1.1(0.7) 0.63(0.4) 0.92(1) 
lcl IcO lc-1 le-3 le-5 le-7 




0/15 
0/15 


fl09 

IPOP-aC 
IPOP-aa 

A /opt 


333 632 1138 2287 3583 4952 
1.1(0.2) 1.2(0.1) 1.2(0.2) 1.2(0.1) 1.1(0.1) 1.1(0.1) 
0.93(0.3) 0.78(0. 2)*%. 78(0. 2)||o. 89(0. 3)* 0.79(0.2) J^0. 75(0. 2) J; 

lcl leO le-1 lc-3 lc-5 le-7 


15/15 fl24 
15/15 IPOP-aC 
15/15 IPOP-aa 
#aucc A/ pt 


192 1959 40840 1.3c5 3.9e5 8.0c5 
1.1(0.5) 3.9(11) 1.0(0.7) 0.91(0.6) 0.76(0.4) 0.62(0.3) 
0.86(0.4) 3.0(0.8) 2.2(2) 1.4(0.5) 0.76(0.3) 4.0(5) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 
15/15 
14/15 
#aucc 


fllO 

IPOP-aC 
IPOP-sa 

A /opt 


00 00 00 00 00 00 
lcl leO le-1 lc-3 lc-5 le-7 


f125 

0/15 IPOP-aC 
0/15 IPOP-aa 
#aucc A/ p t . 


11 1 2.5c7 8.0c7 8.1c7 
1(0) 827(912) 3.8c6(4c6)l.l(l) 1.7(2) 1.7(2) 
1(0) 1083(1766) 3.3e6(2c6p.61(0.5) 1.1(1) 1-1(1) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


4/15 
2/15 
3/15 
#succ 


fill 

IPOP-aC 
IPOP-sa 

A /opt 


00 00 00 00 00 00 
lcl leO le-1 lc-3 lc-5 le-7 


g fl26 

0/15 IPOP-aC 
0/15 IPOP-aa 
#aucc A/ p t 


11 1 00 00 00 
1(0) 6417(3982) 00 
1(0) 7156(4580) 00 

lcl leO le-1 lc-3 lc-5 le-7 




0/15 
0/15 

#3UCC 


fll2 

IPOP-aC 
IPOP-sa 

A /opt 


25552 64124 69621 73557 76137 78238 
0.84(0.2) 0.84(0.5) 0.89(0.5) 0.93(0.5) 0.94(0.5) 0.93(0.4) 
0.89(0.3) 0.79(0.1) 0.84(0.1) 0.88(0.1) 0.88(0.1) 0.88(0.1) 
lcl IcO lc-1 lc-3 lc-5 lc-7 


15/15 

15/15 IPOP-aC 
15/15 IPOP-aa 
#aucc A/ p t . 


11 1 4.4c6 7.3c6 7.4c6 
1(0) 193(102) 2.8e5(4c5)1.0(0.9) 1.1(1) 1.1(1.0) 
1(0) 238(106) 3.3c5(4c5)O.75(0.6) 0.93(0.5) 0.97(0.6) 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 
15/15 
14/15 

#3UCC 


fll3 

IPOP-aC 
IPOP-sa 

A /opt 


50123 3.6c5 5.6c5 5.9c5 5.9c5 5.9c5 
0.60(0.4) 0. 27(0. 1)^30. 27(0. 1)^ 4 0. 29(0. 1)^ 4 0. 29(0. l)^ 4 O.29(0.1)^ 
0.54(0.2) 0. 35(0. 2)^ 2 0. 31(0. 1)^ 4 0. 32(0. 1)^ 4 0. 32(0. 1)^ 4 0. 32(0. 1)^ 4 

lcl leO lc-1 lc-3 lc-5 le-7 


15/15 lLJ - & 
15/15 IPOP-aC 
15/15 Ip OP-sa 

A /opt 
#SUCC 1129 


1.4c5 1.3c7 1.7c7 1.7e7 1.7c7 1.7c7 
0.51(0.6) 0.72(1) 1.0(2) 1.0(2) 1-0(1) 1-0(1) 
1.3(2) 0.59(0.8) 1.1(1) 1-1(2) 1-1(1) 1-1(2) 
lcl leO le-1 lc-3 lc-5 le-7 


9/15 
7/15 
8/15 
#succ 


fll4 

IPOP-aC 
IPOP-sa 

A /opt 


2.1c5 l.lc6 1.4c6 1.6c6 1.6c6 1.6c6 
0.64(0.5) O.35(0.1)^ 4 0.53(0.3)^ 0.62(0.3) 0.62(0.3) 0.63(0.3) 
0.71(0.3) 0. 39(0. 3)^ 3 0. 40(0. 2) ^30.50(0. 3) ^2°- 5 0(0.3)^20- 55 < -3)^ 
lcl leO le-1 lc-3 lc-5 le-7 


15/15 IPOP _ aC 

15 /15 jpOP-aa 
15/15 

A/opt 


7.8c6 4.1c7 4.2c7 4.2c7 4.2c7 4.2c7 
D. 16(0. 2)^ 0.27(0.3) 0.46(0.6) 0.46(0.6) 0.46(0.6) 0.46(0.6) 
0.41(0.9) 0-28(0.5) 0.36(0.5) 0.36(0.5) 0.36(0.5) 0.36(0.5) 
lcl leO le-1 lc-3 lc-5 le-7 


5/15 
6/15 
9/15 
#succ 


#aucc fl3 r 


4904 93149 2.5c5 2.5c5 2.6c5 2.6c5 
1.5(2) 66(99) 54(71) 54(71) 54(72) 53(70) 
0.55(1) 27(34) 30(43) 30(49) 30(39) 30(41) 


7/15 
6/15 
12/15 


fll5 

IPOP-aC 
IPOP-sa 


2405 30268 91749 1.3c5 1.3c5 1.3c5 
1.4(2) 1.2(0.6) 0.54(0.1) 0. 45(0. 0)^ 4 0. 45(0. 0)^ 4 0. 45(0. 1)^, 
0.75(2) 0.67(0.3 ) 0.31(0.2) * 2 4. 8(15) 4.8(15 ) 4.7(14) 


15/15 IPOP-aC 
15/15 IPOP-aa 
15/15 



Table 2: Expected running time (ERT in number of function evaluations) divided by the respective best ERT 
measured during BBOB-2009 (given in the respective first row) for different A/ values in dimension 20. The 
central 80% range divided by two is given in braces. The median number of conducted function evaluations 
is additionally given in italics, if ERT(1CF 7 ) = oo. #succ is the number of trials that reached the final target 
/opt + 1CP 8 . Best results are printed in bold. 
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Figure 3: Bootstrapped empirical cumulative distribution of the number of objective function evaluations 
divided by dimension (FEvals/D) for 50 targets in 10' -8 " 2 ' for all functions and subgroups in 5-D. The "best 
2009" line corresponds to the best ERT observed during BBOB 2009 for each single target. 




Figure 4: Bootstrapped empirical cumulative distribution of the number of objective function evaluations 
divided by dimension (FEvals/D) for 50 targets in It)' -8 " 2 ' for all functions and subgroups in 20-D. The "best 
2009" line corresponds to the best ERT observed during BBOB 2009 for each single target. 



